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Examining How Faculty Reflect on Instructional Data: 
A Call for Critical Awareness and Institutional Support 


Bailey B. Smolarek and Matthew T. Hora 


Reflective practice, or the idea that individuals should critically examine their own 
practices, experiences, and assumptions to improve subsequent decision-making and task 
performance, is a core idea in adult learning and teacher education. Engaging in self-reflection, 
as part of an ongoing process of learning and professional development, is central to theories of 
transformative learning (Mezirow, 1990), self-directed learning (Knowles, 1975), and reflective 
practice (Schon, 1983). While empirical research has been conducted on the nature of reflective 
practice among a variety of professionals in fields such as K-12 education (Patterson, Minnick 
Santa, Short, & Smith, 1993), medicine (Mann, Gordon, & MacLeod, 2009), and social work 
(Sodhi & Cohen, 2012), less research examines the nature of reflective practice among 
postsecondary faculty’ (Lyons, 2006; McAlpine & Weston, 2000). Moreover, much of the 
research on reflective practices of postsecondary faculty only views reflection as an instrument 
for instructional improvement rather than a tool to facilitate instructors’ critical awareness of 
broader socio-political concerns (Zeichner & Liston, 2014). 


This limitation is considerable because a critical awareness of how these issues shape 
teaching and learning is essential to improving student achievement (Gay & Kirkland, 2003). In 
particular, attention to how race, gender, and the unique cultural and socio-economic 
backgrounds of students influence their experiences in college should ideally be part of the 
reflective process (Zeichner & Liston, 2014). However, issues of equity and diversity are not 
usually primary considerations among postsecondary instructional designers and faculty at the 
(Harper, Patton, & Wooden, 2009; Miyake et al., 2010). Furthermore, because most faculty have 
never received formal teacher training, they have a tendency to uncritically replicate teaching 
approaches they experienced as students themselves regardless of efficacy (Mazur, 2009; Oleson 
& Hora, 2014). These issues collectively point to the need for faculty to critically reflect on their 
teaching practices as well as on specific organizational, cultural, and political contexts of 
instruction (Hora, 2016; Lattuca & Stark, 2011). 


However, today’s discussions about reflective practice in higher education are unfolding 
in a unique political and technological context that must also be taken into account. Spurred on 
by the accountability movement and advances in instructional technology and analytics, the data 
driven decision-making (DDDM) movement has gained prominence in both K-12 and higher 
education based on the idea that decisions about instructional design and classroom teaching 
should be informed by numeric evidence and empirical research on teaching and less by 
instructors’ anecdote, hunch, or intuition (Lane, 2014; Marsh, Pane, & Hamilton, 2006). A 
central idea of DDDM is that reflective practice can be initiated once teachers are confronted 


1 By faculty we mean all people who hold teaching positions—whether full- or part-time, in a tenure track—in 
postsecondary institutions. 
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with data about student achievement and the relative efficacy (or lack thereof) of their own 
teaching, whereupon reflection and improvements to teaching practice will follow (Mandinach, 
2012; Wieman, Perkins, & Gilbert, 2010). 


But research on DDDM in K-12 districts and schools has shown that the provision of data 
alone does not magically transform practice; rather, educators interpret data based on a 
combination of their personal experiences and routinized practices and cultural norms within 
their organizations in idiosyncratic and sometimes ineffectual ways (Coburn & Tumer, 2011; 
Spillane, 2012). Furthermore, studies on DDDM in higher education reveal that faculty and 
administrators rely on a variety of information beyond the typical conception of data (i.e., 
numbers) and instead draw upon information such as verbal conversations with colleagues and 
students as part of their reflective process while paying close attention to contextual factors 
(Foss, 2014; Hora, Bouwma-Gearhart, & Park, in press). 


Consequently, research on reflective practice among postsecondary faculty needs to take 
into account the use of multiple data forms as well as the involvement of critical awareness of 
the socio-political conditions of learning. In this paper we report findings from an interview- 
based study exploring these issues among 21 instructors at three California research universities. 
Using a structured approach to grounded theory (Charmaz, 2006; Strauss & Corbin, 1994) we 
found that faculty drew upon a variety of data (e.g., grades, colleagues, non-verbal student 
feedback) to engage in three distinct types of reflection: instrumental (i.e., rapid assessment of 
technical issues), structural-critical (i.e., in-depth assessment of institutional issues affecting 
teaching/learning), and social-critical (i.e., in-depth assessment of underlying factors influencing 
student outcomes), all of which were strongly influenced by the local organizational and socio- 
cultural context. 


In addition to providing new evidence regarding the nature of reflective practice among 
faculty, our study contributes to the understanding of reflection itself by interpreting results using 
two analytic frameworks not commonly used in the postsecondary literature. We draw upon 
culturally responsive frameworks from K-12 education (Gonzalez, Moll, & Amanti, 2013; 
Ladson-Billings, 1995; Villegas & Lucas, 2002) as well as insights from dual process theory in 
psychology (Evans & Stanovich, 2013) to interpret the data. Ultimately, we found that although 
faculty in our sample drew on similar forms of teaching related data to regularly engage in 
reflective practices, the subsequent outcomes of this reflection was severely limited by both 
individual bias and institutional constraints. Therefore, we argue that in order to substantively 
change implicit biases, spark professional growth, and promote student learning, a more holistic 
notion of “data” that recognizes the value of non-numeric information is needed. Likewise, we 
urge faculty to examine these data with a more critical eye, particularly with respect to the 
learning outcomes of diverse and historically underrepresented student populations. Finally, we 
argue that adult education institutions play a significant role in facilitating this critical 
examination and must provide faculty the necessary space, time, and resources to engage in 
critical reflection as well as the appropriate institutional mechanisms to voice concerns and enact 
change. 
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Background 


Many regard John Dewey (1933) as the first U.S. educational theorist to view teachers as 
reflective practitioners. For Dewey, reflection signified a systematic and disciplined process of 
thoughtful, open-minded inquiry that subsequently ended in a sustainable conclusion. However, 
some scholars (Fendler, 2003; Jordi, 2011) have pointed to Cartesian rationality, where objective 
self-awareness is seen as a valid way to generate knowledge, as the root of reflectivity. 
Ironically, the more recent revival of reflective practice came in the 1980s through a critique of 
this objectivist perspective (Schén, 1983). Following this revival, scholars and practitioners from 
a variety of disciplines began to take up the notion of reflection. The field of education, in 
particular, has widely embraced reflection and has come to a general consensus that all teachers 
should be reflective. Still, the precise definition of “reflection” remains unclear and highly 
contested (Zeichner & Liston, 2014). Because of this, we provide Table 1 to summarize the 
scholarly conceptualizations of reflection that we drew upon to inform our work. Following this 
table, we discuss four ideas within the reflective practice literature that played key roles in our 
analysis, and we describe prior work on reflective practice conducted in higher education. 


Table 1. Scholarly Conceptualizations of Reflection 
Scholar Concept 


Schon, 1983 Reflection is the cyclical act of thinking about what one is doing, during 
or following the action, to facilitate the learning process. 


Mezirow, 1990 The act of reflection allows space for investigating one’s beliefs and 
engage in problem-solving. 

Brookfield, 2000 Reflection is different from critical reflection. Reflection focuses on 
technical decision-making while critical reflection is rooted in critical 
theory and examines power dynamics and embedded assumptions. 


Fendler, 2003 The distinction between “critical reflection” and “reflection” is false 
because even critical practitioners may engage in technical reflection if 
“they believe that the efficient mastery of subject matter by their students 
is the most effective means of redressing social inequities” (p. 21). 


Jordi, 2011; Sodhi | Reflection has been conceived of too narrowly as only cognitive and 


& Cohen, 2012 needs to be expanded to more “embodied” forms of knowledge. 

Zeichner & Reflection is a collaborative process that enables teachers to examine 

Liston, 2014 their thoughts and actions to work toward a more socially just world. 
Tacit Knowledge 


In 1983, philosopher and organizational theorist Donald Schén wrote The Reflective 
Practitioner in response to the dominant view of professional knowledge and practice as overly 
specialized and technical. In contrast, Sch6n argued that in practice, problems are frequently ill- 
defined and contain factors the problem solver must interpret in real-time. He argued that 
competent practitioners utilize what he called knowledge-in-action, a form of tacit knowledge 
that over time becomes an automatic response, to solve in-the-moment situations and problems. 


How Faculty Reflect on Instructional Data 


He also argued that when tacit expertise is insufficient, the problem-solver should reflect in the 
moment (reflection-in-action) or afterward (reflection-on-action) to assess the decision and 
adjust practice accordingly. Educational researchers have built on Sch6n’s ideas by emphasizing 
the importance of the practical wisdom from school leaders (Halverson, 2004) and classroom 
teachers (Zeichner & Liston, 2014). This approach indicates the importance of viewing the 
everyday experiences and tacit knowledge of teachers as significant sources of “data” in 
improving student learning while still recognizing the need to interrogate that knowledge through 
reflection. 


Critical Reflection 


Many scholars (Brookfield, 2000; Larrivee, 2000; Mezirow, 1990; Zeichner & Liston, 
2014) distinguish simple “reflection” from “critical reflection.” However, while most agree that 
critical reflection is the deliberate examination of one’s underlying assumptions and beliefs, 
scholars disagree on the ultimate purposes of such a reflective practice. For some (Brookfield, 
2000; Zeichner & Liston, 2014) the purpose of critical reflection is to lead one to a greater 
understanding and promotion of emancipation, empowerment, and social justice within 
educational contexts, while others emphasize the elimination of bias to attain a more objective 
and self-aware perspective (Mezirow, 1990). We situate our understanding of “critical 
reflection” with Brookfield’s and Zeichner’s perspectives as we see the purpose of critical 
reflection is to work toward a more equitable and inclusive educational environment. 


Role of Context, Emotion, and the Body 


Another critique of the literature on reflective practice is the predominant focus on 
reflection as a cognitive or “in-the-head” phenomenon that overlooks the role of context, 
emotion, and the body itself. Alternatively, some have proposed more “embodied” perspectives 
that explore forms of meaning-making that implicate the body as a site of knowing (Jordi, 2011; 
Sodhi & Cohen, 2012; Tremmel, 1993). For example, Sodhi and Cohen (2012), in an interview- 
based study of 10 social workers, found that participants often relied on physical manifestations 
of knowing with the stomach (i.e., “gut reaction”), breath, and fatigue as primary areas of focus. 
They concluded that the body served as a source of feedback likening physical symptoms with 
Schén’s reflection-in-action as an indicator for needed reflection. 


Insights from Cognitive Science on Reflective Practice 


Insights from the cognitive sciences have also shed light on the processes that may underlay 
reflective practice itself. First, dual-process theories of cognition posit that the brain processes 
information in two distinct ways: the peripheral route (i.e., Type 1 processing) is fast, automatic 
and involves cognitive heuristics to arrive at decisions with little effort, whereas the central route 
(i.e., Type 2 processing) involves higher order reasoning that is slow, logical and deliberate 
(Chaiken & Trope, 1999; Evans & Stanovich, 2013). Reflection is decidedly a Type 2 process, 
evident in neuroscience studies where reflection implicates different parts of the brain 
(prefrontal, frontal cortical) than Type 1 (limbic system). While Type 1 can draw on expertise, 
cognitive heuristics developed through countless hours of practice and experience, it can also 
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include implicit biases and incorrect assumptions (Kahneman, 2011). In an analysis of how 
these ideas may apply to issues of educational reform, Gregoire (2003) argues that when reform 
messages are seen as not implicating the self, a Type 1 process of decision-making ensues. Thus, 
change in one’s teaching practices and/or transformation in one’s underlying beliefs about 
teaching and learning requires a Type 2 process where the decision-maker views him or herself 
as part of the problem and subsequently engages in a sustained period of self-examination. 


Reflective Practice among Postsecondary Educators 


While the literature based on reflective practice among postsecondary faculty is limited, 
several studies do shed light on its prevalence and characteristics. Early work exploring this topic 
included Biggs (1999), and McClean and Blackwell (1997) who argued that “teaching excellence 
resides in a reflective, self-critical” approach (p. 85). Kane, Sandretto, and Heath (2004), in their 
investigation into the practices of 17 award-winning faculty, found a “common 
characteristic...was that these excellent university teachers engaged in regular, purposeful 
reflection on their teaching practice” (p. 300). Kane, Sandretto, and Heath classified their 
participants’ types of reflection their participants engaged in as technical (i.e., decisions about 
immediate issues), descriptive (i.e., analyzing and explaining performance), dialogic (i.e., 
considering alternatives for action), and critical (i.e., considering effects of self on others, socio- 
political forces at work). They found that faculty most frequently engaged in technical (16 
faculty) and descriptive reflection (16) followed by dialogic (13) and critical (3), with critical 
reflection referring to reflection that considers the social, political, and historical context. Finally, 
Lyons (2006) advanced the definition of reflective engagement by exploring the use of reflective 
teaching portfolios by university faculty. Beyond these studies, however, little empirical work 
exists on the prevalence of reflective practice among postsecondary faculty. 


Researchers have also focused on the importance of organizational contexts and 
situations in shaping how faculty engage in reflective practice, such as insufficient time (Kuit, 
Reay, & Freeman, 2001) and de-motivated, overworked staff (Davis, 2003). Likewise, McAlpine 
and Weston (2000), in their study of the reflective practices of six “successful” university 
professors, found that improving one’s teaching requires a desire to teach, recognition of and 
motivation to develop pedagogical knowledge, a supportive environment, minimal constraints, 
opportunities for practice, and formal teacher training. These features highlight the individual 
and institutional characteristics needed to improve teaching. Still, more research is needed to 
better understand the nature of reflective practice, particularly critical reflection, in this 
population, especially in the current climate of accountability that centers on numeric data. 


Therefore, given the importance of deliberate, reflective practice in improving 
postsecondary teaching and learning, it is important to understand how faculty are actually 
engaging in reflection. Moreover, we need to understand not only what faculty do in terms of 
reflection, but how and why they do what they do. Descriptive insights into these processes not 
only increase the ecological validity of observations, but also shed light on the factors that may 
act as “leverage points” that appear to be supporting or inhibiting particular practices (Coburm & 
Turner, 2011; Spillane, 2012). In an era of increasing pressures for accountability and mandates 
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for instructional change and improvement, such insights can be an important tool to improve 
interventions by ensuring that they align with or respond to the norms and practices of specific 
organizations, as opposed to a “top-down” approach that is a far less effective approach to 
reform (Fullan, 2010; Spillane, Halverson, & Diamond, 2001). 


Methods 


Sampling and Data Collection 


This study was conducted at three large, public Research 1 universities in California to 
investigate faculty use of instructional data. The sites were selected on the basis of similarities in 
their geographic, political, and socio-economic settings. Two of the study sites were campuses in 
the 10-campus University of California (UC) system, and one was in the 23-campus California 
State University (CSU) system. It is important to note that the CSU campus was one of two 
polytechnic institutions in the system. We are referring to the two UC institutions as UC San 
Miguel and UC Beachfront and the CSU institution as Cal State Parkside. 


Another important part of the study context is that the UC and CSU systems have been 
significantly affected by budget cuts over the past several years. These cuts have led to increased 
workloads for instructional staff, more courses taught by contingent (i.e., non-tenure-track) 
faculty, less funding for graduate assistants, and more online programming and instruction. In 
fact, according to the 2015 UC Accountability Report, “UC’s student faculty ratio is at the 
highest level it has ever been and is also high relative to research universities of comparable 
quality” (p. 129). Further, in the UC system both the student to faculty ratio (from 19:1 in 2004 
to 21:1 in 2014) and the amount of student credit hours taught has increased. These cuts, 
however, are not unique to California, which makes investigation of faculty decision-making and 
reflective practices during these uncertain times of budget struggles, reduced state support, and 
increased institutional constraints on teaching activities all the more important. 


The goal for the larger study upon which this paper is based was to capture DDDM 
processes in science and engineering disciplines, given the focus of the funding agency for the 
study (the National Science Foundation). At UC San Miguel, four science disciplines were 
included (biology, physics, chemistry, geoscience) as no engineering department existed at this 
campus. To compensate for the lack of engineering participants at this study site, engineering 
departments at the other two sites were the principal focus. 


A non-random purposeful sampling strategy was used to recruit faculty and administrator 
participants for the study. The sampling frame for instructors was compiled using each 
institution’s course catalogue for the Winter 2015 term, and for administrators using institutional 
websites for salient colleges and departments. Of the 115 instructors who were contacted via 
email to participate in the study, 21 instructors and nine administrators ultimately participated 
(an 18% response rate). Only the 21 faculty participants were included in this analysis, who 


? All school and instructor names are pseudonyms to protect confidentiality. 
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represent the following disciplines: biology (n=5), physics (n=1), chemistry (n=2), geoscience 
(n=1), mechanical engineering (n=8), and industrial and manufacturing engineering (n=4). 
Demographically, these instructors were predominately white males, with about one-third of the 
participants being female. This situation is important to understand because while this sample is 
comparable to STEM faculty across the country (UC Davis Advance, 2015) it highlights the lack 
of diversity among faculty in these departments. Finally, of the 21 instructors, 13 were full-time 
professors while six were lecturers and two were post-doctoral fellows. Thus, work experience, 
course loads, and additional benefits and responsibilities varied among participants. 


A team of three researchers? conducted all data collection activities during the Winter 
term of 2015 (January-March). Data collected for this study included interviews and documents 
from institutional webpages, accreditation reports, and minutes of internal meetings. A Critical 
Decision Making approach was utilized for the interviews, which included questions focused on 
a single decision and/or activity, followed by in-depth probes that elicit respondent accounts 
about their thought process regarding decision steps and considerations of influential factors 
(Crandall, Klein, & Hoffman, 2006). Respondents were asked about their most recent instance of 
reflecting upon teaching-related data, their reasons for doing so, the nature and origins of the 
data, and the outcomes of the reflection process. Follow-up questions elicited information about 
other salient types of data or information used in their overall course planning and/or teaching 
activities, and specific policies within their institutions that encouraged the use of teaching- 
related data. Interviews took place in respondents’ offices or nearby conference rooms and lasted 
approximately 45 minutes. 


Data Analysis 


All interviews were transcribed and entered into NVivo qualitative data analysis 
software, whereupon two analysts segmented the raw data into smaller, more manageable units. 
The segments pertained to three core topics central to the study: individuals’ data routines, the 
existence of continuous improvement systems, and contextual factors related to data use. Data 
segments that contained references to reflective practice were analyzed using a structured 
approach to grounded theory, which involves the inductive analysis of data while considering the 
research questions and external theoretical frameworks (Charmaz, 2006; Strauss & Corbin, 
1994). The analysis began with an open-coding process, where codes were created from the data 
and each successive instance of a particular code was compared to previous instances of that 
code (i.e., the constant comparative method) (Glaser & Strauss, 1967). Additionally, throughout 
the coding process, analytic memos were written to assist the development of a coding system as 
well as to document moments of insight regarding patterns or themes in the data (Saldafia, 2013). 
Analysis focused on references to the data participants reported using as part of their reflection, 
the reflection process participants described, and the results participants reported from their 
reflection. From this process, themes emerged that elaborated on the meaning of codes’ 


3 Matthew T. Hora (University of Wisconsin—Madison), Jana Bouwma-Gearhart (Oregon State University), 
Hyoung-Joon Park (University of Wisconsin—Madison) 
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“truncated essence” (Saldafia, 2013). All codes and themes were discussed at length between the 
researchers to ensure reliability. 


Limitations of the study include a small, self-selected sample at specific types of 
postsecondary institutions such that generalization to larger populations of faculty within and/or 
beyond the study institutions should not be made. An additional limitation is the reliance on self- 
reports about reflective practice on teaching-related data that may or may not reflect 
respondents’ actual behaviors. Consequently, we encourage researchers to build upon our results 
by conducting research with larger samples at diverse institutional sites, and by using techniques 
such as think-aloud tasks to capture reflective practices as they unfold in real-time. In addition, 
future research should focus on the nature of reflective practice among other types of adult 
educators, particularly those working outside of large university settings. 


Results 


Data Used in Reflective Practice 


When asked what types of data they regularly reflected upon, faculty discussed six 
distinct categories of data ranging from exams to conversations with students (see Table 2). Four 
of these categories are non-numeric sources of information that may not be thought of as “data” 
in most of the DDDM literature. However, we consider these sources valid and important 
because respondents considered them to be key to their reflective processes. This emphasis 
points to the need for proponents of DDDM to broaden their understanding of “data,” which is 
also supported by information scholars who argue for the inclusion of non-numeric forms of 
information within complex organizations because of their salience and importance to 
organizational actors (Pentland, 1995). 


Table 2. Types of Data used in Reflection 


Type Definition 

Grades Students’ course grades or scores on assignments and exams 

Formal evaluations Institutional student course evaluations or peer evaluations 

Research literature Pedagogical literature, teaching blogs, or professional development 
materials 

Colleagues Communication with peers within the university, colleagues who 


teach the same discipline at other universities, or university 
academic learning coaches 


Direct student Direct conversations or communication with students 

feedback 

Non-verbal cues Students’ physical reactions during class (e.g., bored or engaged 
expressions) 
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Of the types of data faculty reported reflecting upon, all respondents listed examples of 
exam scores, homework grades, and formal student or peer evaluations. However, many found 
institutional course evaluations to be insufficient and unhelpful, and took it upon themselves to 
create their own evaluations that were administered either mid-semester or at the end of the 
course. Several respondents also reported reflecting upon the research literature and online 
resources such as teaching blogs. Additionally, some instructors noted that discussions with 
colleagues provided important information and insights that were then reflected upon as part of 
their efforts to become better teachers. 


Interestingly, direct student feedback was arguably the most highly valued type of 
information used during reflective practice. Many instructors went out of their way to elicit 
student feedback through informal surveys and conversations. For example, Professor Narayanan 
(UC Beachfront) stated that she heavily relied on conversations with students and her own 
teaching assistants. She stated: 


Talking to them [the students] and getting information out of them really helps... I spend 
a lot of time in conversations with students about, ‘What do you think about the class so 
far?’...and every week I ask my TAs, ‘Okay, what did they learn? What did they not 
learn? What are the students getting? Did you ask them what they think?’ 


Similarly, Professor Walsh (UC San Miguel) claimed, “One of the most important sources (of 
information) are students just conveying their thoughts directly.” These sentiments underscore 
the importance of personal communication between teachers and students in the reflective 
process. 


However, some instructors observed that students whose voices were loudest, meaning 
the ones who volunteered feedback in the classroom or who came to office hours, were the ones 
whose comments were most likely to be heard and reflected upon. For example, Professor 
Sullivan (UC San Miguel) noticed that her office hours were typically attended by a small group 
of the same students, observing that, “It’s a self-selected group of students, so one difficulty is 
that with a class of 200 students there are many students that I never get to know or talk to unless 
they are willing to put themselves up and say ‘I need help.’” Therefore, the concerns of 
struggling students outside of this self-selected group may go unnoticed unless students directly 
communicate with their instructors. 


In addition, we encountered two examples of what Sodhi and Cohen (2012) refer to as 
“embodied knowing,” or information that takes a physical manifestation in the body. Professor 
Bowden (UC Beachfront) and Professor Bauer (UC San Miguel) discussed examples of physical 
reactions students have in the classroom as a way of knowing whether their students are learning. 
For example, Professor Bauer described an important type of information as, “The immediate 
reaction of students when you see the lights go on.” This type of nonverbal communication was 
also discussed by Professor Bowden, who claimed that the most meaningful information that 
informed his teaching was “the reaction of the students” and whether they were bored, interested, 
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or paying attention. Thus, even though only two of our 21 participants discussed this type of 
knowing, we still reported it as playing a significant role in reflective practice. These examples 
speak to the idea that reflection cannot be conceptualized as a purely cognitive act. 


Types of Reflective Practice 


In our analysis of faculty’s reflective observations, we drew upon critical reflective 
practice theories (Brookfield, 2000; Mezirow, 1990; Zeichner & Liston, 2014) to interpret our 
data and distinguish among three distinct types of reflection. First, instructors discussed 
reflections focused on instrumental or technical issues related to their courses, such as textbook 
or syllabi issues. Second, instructors also reported a more intensive reflection process that we 
call structural-critical, whereby structural issues such as class sizes were considered in a critical 
light as one of several factors influencing teaching behaviors and student outcomes. Third, 
instructors discussed issues related to the socio-cultural and economic contexts of students’ lives, 
a reflective process we call social-critical. The following sections discuss in depth these three 
types of reflective practice. 


Table 3. Types of Reflective Practice 


Type Definition Example 

Instrumental Reflection that centered on Too much text on PowerPoint slides 
technical teaching tasks 

Structural- Reflection that centered on Large class sizes 

critical institutional issues that affected 
teaching and learning 

Social-critical Reflection that critically examined | Historically underrepresented 
sociocultural issues that affected student populations doing more 
teaching and learning poorly in class than dominant 

populations. 


Instrumental reflection. Most respondents discussed their reflective practice in terms of 
technical issues related to their course curricula and/or instructional strategies that were 
ultimately altered or corrected. The majority of these changes would be considered “technical” 
(Brookfield, 2000; Zeichner & Liston, 2014), “instrumental” (Mezirow, 1990), or “practical” 
(McAlpine & Weston, 2000) by reflective practice scholars. For example, several participants 
reported lecturing less often while others reported trying to vary course topic order after a period 
of reflection. In this study, we are referring to this sort of everyday decision-making to improve 
isolated classroom practices and learning as instrumental reflection. 


The majority of instructors’ instrumental reflection resulted in discrete classroom changes 
instructors individually made. For instance, if all students in the course consistently struggled 
with a particular exam question, the instructor would alter his or her instruction to spend more 
time discussing the topic pertaining to the exam question. Respondents consistently reported 
ways they personally had made instrumental changes with examples ranging from “I changed the 
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way I lay[ed] out my course” (Professor Wagner, Cal State Parkside)” to “I have learned to 
speak louder [and] have less text on my slides” (Professor Sullivan, UC San Miguel). 


Structural-Critical and Socio-Critical Reflection 


While the everyday troubleshooting involved in instrumental reflection is an essential 
part of teaching, and illustrates the time and concern these instructors have for their classes, 
teachers are not “mere technicians” (Zeichner & Liston, 2014, p. xvii). Teaching is situated is a 
complex web of broader contexts ranging from institutional norms to the lived experienced 
learners and instructors bring to the classroom. Thus, instrumental reflection, alone, cannot 
account for the larger power dynamics that ultimately affect learner achievement. In fact, several 
scholars (Brookfield, 2000; Mezirow, 1990; Zeichner & Liston, 2014) have distinguished 
instrumental reflection from other forms of more “critical” reflection that require deeper 
consideration. Many of our instructor participants did engage in such reflection, which pushed 
beyond the instrumental and towards more structural and social concerns. In this study, we refer 
to this sort of reflective practice as structural-critical and socio-critical reflection. 


Structural-critical reflection. Several respondents discussed structural issues within 
their institutions that appeared to influence their teaching strategies and/or student outcomes. The 
primary structural issues faculty discussed included large class sizes, a lack of time to “cover” 
the content in a course, and ineffectual online learning systems. While these factors may not be 
considered data by most of DDDM literature, they are included in the analysis because 
respondents considered them to be important variables that shaped how they interacted with 
students and engaged in their work as educators. However, instead of directly trying to address 
these problems, instructors either accepted them as givens or sought coping mechanisms to 
“make do” with what they had. These concerns draw attention to the limits of reflective practice 
as well as to the institution’s role in student success. 


Through their reflection, several instructors noted classes were often too big, creating an 
inevitable distance between students and instructors. Instructors lamented at not being able to 
learn all of the students’ names, get to know students individually, or assist all students with 
coursework and projects. In response, some instructors sought alternative ways to manage large 
classrooms and promote student engagement. Professor Walsh (UC San Miguel), for example, 
explained that despite not knowing all of his students’ names, he still tried to connect with 
students by “Just throw[ing] out questions in a big hall or ... randomly pick[ing] someone...to 
get them engaged.” Professor Narayanan (UC Beachfront), on the other hand, prided herself on 
memorizing the names, and even the handwriting, of all 75 of her students even though it was 
rather difficult. Finally, Professor Altman’s (UC San Miguel) strategy was to group students into 
pairs, saying, 


What really drove [putting them in groups] was that the enrollment in the class doubled 
and it just became impossible. The issue is really the final projects. All the students do 
final projects. When there are 25 students in the class, it’s possible to supervise 25 
projects. Once there are 45 students, it’s impossible. 
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Thus, even though numerous instructors noted the issue of crowded classrooms, none of 
them reported trying to directly affect enrollment policies and practices. Instead, they “made do” 
by trying to manage in other ways. 


Faculty also discussed not having enough time to sufficiently cover all required course 
materials or provide necessary review. Several respondents noticed that there were often topics 
that some or even all students struggled to grasp, but because of time constraints they were 
unable to allocate additional class time to these subjects. In response, some instructors coped 
with this problem by offering copies of their lecture slides before class. For example, Professor 
Walsh (UC San Miguel) stated that he, “Can’t do anything about time, [but] I can do something 
about providing lecture slides.” Thus, through reflection, instructors recognized time constraints 
as a Salient issue affecting student learning, but were only able to enact minor workarounds to 
address the problem. Likewise, faculty also reported that the lack of time affected their ability to 
prepare lessons, update course curricula, or experiment with new instructional strategies. For 
example, Professor Sullivan (UC San Miguel) stated that she would like to experiment with 
active learning techniques but that it “requires time I don’t have.” 


Additionally, some instructors noted that certain online learning systems seemed to be 
burdensome and ineffective, but they were unable to stop using the system and could only try to 
troubleshoot as best as possible. For example, Professor Douglas (Cal State Parkside) claimed 
that the online learning management system at his institution “hinders” student learning because 
it is “confusing.” As a result, he reported spending valuable instruction time explaining to his 
students how to use this system. Likewise, Professor Mulder (UC San Miguel) stated that the 
biggest student complaint from the previous year was the web assignment system. He observed 
that they began using the online automated system because of cuts to graduate teaching assistants 
who used to do most of the grading. The biggest problem with the system was that if students 
“don't type [the answer] in correctly, it can register as wrong,” which has led to students being 
extremely frustrated with the system. Even though students identified this problem as a 
significant concern, Professor Mulder admitted that he “can’t help them too much on it” besides 
providing a list of common word errors. Thus, even though both students and faculty noticed 
significant, practical flaws with the online system, institutional constraints prevented them from 
discontinuing their use. 


One of the commonalities among these structural-critical forms of reflection was that 
they involved the critical consideration of organizational factors that inhibited instructional 
innovation and/or student success. Yet, despite engagement in such reflection, faculty ultimately 
felt nothing could be done about structural issues besides minor workarounds. Consequently, the 
reflective process resulted in the consideration of new information or data, but the outcomes of 
the reflection were limited to identifying creative solutions to work around these factors instead 
of addressing the root cause of the situation. This state of affairs draws attention to the limits of 
reflective practice as well as to the considerable role organizational context plays in shaping 
teaching behaviors and student experiences in the classroom (Hora, 2016). 
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Social-critical reflection. After instructors had made instrumental classroom changes 
and developed workarounds based on their structural reflection, some students struggled. 
However, instead of attempting additional action or changes to assist those students, faculty 
rationalized the situation through the notion of an “acceptable failure rate.” This idea speaks to 
larger social issues that center on embedded assumptions instructors hold about students and 
about student failure and success in general. Several instructor participants openly expected that 
there would always be students who struggled, were unmotivated, or unprepared. This 
expectation means that instead of interrogating the numerous reasons for why a student is not 
succeeding (i.e., home issues, work commitments, health problems, pedagogical disconnects), 
instructors accepted student failure as an individual problem that inevitably occurs with a certain 
percentage of students. These issues speak to the need to shift the reflection paradigm from 
focusing on instrumental issues to more critical reflection (Brookfield, 2000; Zeichner & Liston, 
2014) to better support learners and promote equitable education. 


Although all instructors listed numerous instances of using student exams scores and 
direct feedback as part of their reflective practice, they also provided other instances where that 
data was discredited because of biases toward students and learning. For example, several 
instructors discussed the notion of sifting through reliable and unreliable student feedback. This 
meant that instructors found feedback from students who were doing reasonably well in the 
course to be reliable while feedback from failing students was discredited. For instance, 
Professor Kuzmich (UC San Miguel) claimed that in an introductory class of 300 to 400 mostly 
first-year students, 


There is a percentage that just fail miserably for nothing to do with what you’ve done. 
They don’t come to class, they don’t stay, they don’t prepare, they don’t take it seriously, 
and you got to realize that you can’t factor those people into feedback. 


He went on to explain that although he and his teaching assistants try to discern which 
students are “truly” struggling from those who just are not trying, he still believed his “12 to 15 
percent” failure rate was “acceptable.” As for the students deemed failing due to lack of effort, 
he claimed that they were there only “because they don't know what else to do with their lives,” 
so “you can't be too wrapped up with wasting energy” on them. Likewise, Professor Mulder (UC 
San Miguel) discussed how he made sense of students perpetually failing as “some people have 
struggled so much that [no] amount of either effort they’re putting in or resources we would have 
to put into that person” will help because “some people just aren't going to be good scientists.” In 
these cases, instead of considering the numerous reasons why a student was not succeeding, 
instructors accepted student failure as an individual problem that inevitably occurred with a 
certain percentage of students. Therefore, while student motivation may certainly have affected 
learning, faculty dismissed the role of instructors to engage students and investigate underlying 
reasons for a lack of student achievement. However, this situation begs the question whether 
faculty have the structural support, time, and pedagogical tools to engage in such an 
investigation. 
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In fact, the only instance where the idea that student failure may be caused by forces 
external to the student was brought up by Professor Han (UC Beachfront), who framed it as an 
excuse rather than a legitimate explanation. He stated, 


Some people [students] think it’s just too much work and they’ ve got other stuff going 
on. They’ve got jobs, they compete in sports... And if they do well, they love it. They get 
an A on the midterm, they love the course! ... But for me, I’ve done this a few times so I 
kind of know when to take some comments seriously and when not to. 


The discrediting of certain students’ experiences frames learning as a personal 
responsibility dependent primarily (if not completely) on the individual’s own efforts and 
motivation. However, education researchers have long argued that learning and academic 
success is much more complicated than that as it involves numerous socio-cultural and structural 
factors, such as poverty, race/ethnicity, peer networks, and language differences that affect 
student outcomes (Anyon, 1997; Bank & Banks, 2003; Gonzalez et al., 2013; Ladson-Billings, 
1995; Villegas & Lucas, 2002). This complexity is particularly true for first-generation students 
or students of color who do not generally have the same financial or social capital as their white, 
middle class peers and who may be more likely to hold out-of-school jobs or responsibilities. 


However, one professor in our study sample challenged the dominant narrative of an 
acceptable student failure rate. Professor Sullivan (UC San Miguel) highlighted issues of 
educational disparities among traditionally underrepresented populations at her institution. She 
explained her department had begun collecting student performance data in introductory courses 
to notify students whether they were on track to completing the major in four years. The 
department developed various metrics to predict later success in the major. However, she noted 
that, “the downside is that those metrics differentially affect underrepresented students,” which 
goes against the department’s efforts to increase the proportion of these students in their 
programs. Therefore, although neither Professor Sullivan nor her colleagues have come to any 
firm solutions, she reported trying to use more “active learning techniques” because she had read 
that “it helps underrepresented students.” She also engaged in more critical reflection on the 
topic by asking the question, “why historically, have our underrepresented students performed 
poorer,” suggesting that the poor educational outcomes may have something to do with uniform 
teaching styles, especially extensive periods of didactic lecturing. In this way, through her 
reflective practice, Professor Sullivan made a connection between failure rates and historically 
underrepresented populations, while considering the cause was not due to individual student 
deficits but could instead reside within the instructional practices of the faculty. 


Discussion 


Critics of postsecondary education in the United States argue that faculty must change 
how they design and teach their courses for a variety of reasons: Students are simply not learning 
enough (Arum & Roksa, 2011), the nation needs more critically thinking (Bok, 2009; President’s 
Council of Advisors on Science and Technology, 2012), and the college experience plays a 
strong role in determining the ultimate success (or lack thereof) of underrepresented students 
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(Miyake et al., 2010). At the heart of many strategies to effect improvements in educational 
practice is the notion, fueled in part by the DDDM and learning analytics movement, that the 
provision of numeric data will transform practice by sparking a process of analysis, introspection 
and eventually, behavior change (Lane, 2014; Mandinach, 2012). 


Yet the data we report in this paper support prior work in K-12 settings indicating that 
data use is a complex and varied process shaped by cognitive, cultural, and contextual factors at 
the local level (Coburn & Tumer, 2011; Spillane, 2012). Further, the data also support earlier 
work on faculty reflection that found among university faculty a lack of an important type of 
reflection: critical reflection that involves intentional and substantive questioning of one’s own 
practice and the socio-political conditions of learning (Kane et al., 2004). Our findings reveal 
that a group of faculty in three California research universities drew on a wide-range of teaching 
related data and information to inform reflective processes that were primarily technical in 
nature, with subsequent solutions delimited by contextual factors such as workload, resource 
availability, and class size. 


In the remainder of the paper we first present a descriptive model of our findings and then 
discuss these findings and extend previous literature on the topic by drawing on insights from 
culturally responsive K-12 education frameworks and dual process theory to interpret the 
situated nature of critical reflection in postsecondary institutions and implications of the absence 
of critical reflection related to issues of equity, diversity, and student achievement. 


Model of Postsecondary Faculty Reflective Practice 


Our descriptive model (Figure 1) of our findings that advances a new approach for 
thinking about reflective practice in colleges and universities. Instead of an unproblematic and 
simple process of mental reflection on numeric data, this real-world model demonstrates how 
reflection implicates a myriad of steps, processes, and contextual factors such that reflection is 
best thought of not solely as a cognitive “in-the-head” activity (Jordi, 2011) nor as an 
uncomplicated process wherein numeric data can magically transform practice (Spillane, 2012). 
Rather, our model of postsecondary faculty reflective practice involves a variety of information 
types, distinct processes of mental activity, and contextual factors that bound the nature of 
reflection itself and the nature of solutions that result from reflective practice. 
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Figure 1. Cycle of Postsecondary Faculty Reflective Practice 
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The model begins with the types of data faculty reported reflecting upon. Data sources 
not only include exam scores and formal evaluation reports, which would typically be considered 
data in conventional accounts of DDDM, but also social, contextual, and embodied forms of 
information. After collecting or recognizing these forms of information, faculty engage in 
cursory or in-depth reflective processes that result in what we have categorized as three types of 
reflection (technical-instrumental, critical-structural, and critical-social). The subsequent actions 
that faculty take after reflection is significantly connected to these types of reflection. 
Instrumental reflection tends to result in changes to technical aspects of classroom teaching, 
structural reflection results in discrete classroom workarounds to deal with broader structural 
issues, and social reflection results in a rationalization of persistent student failure rates. 
Importantly, the subsequent action faculty take upon reflection is not the end of the reflective 
practice. Rather, faculty may return back to the reflective practice data after taking action and 
begin the process anew in the spirit of continuous improvement. Furthermore, this process does 
not unfold in a vacuum but is shaped by interpersonal relationships as well as unique conditions 
of the organization, department and even individual classrooms. This model of how faculty 
engage in reflective practice not only provides an account grounded in real-world practices and 
perspectives, but also broadens the field of higher education’s understanding of what “data” are 
and the nature of reflection itself as a multi-faceted process. 


Promoting Critical Reflection and Structural Change in Postsecondary Education 


Although our findings indicate that faculty engaged in Type 2 processing (Chaiken & 
Trope, 1999; Evans & Stanovich, 2013) to reflect on deeper issues, they also suggest a lack of 
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critical reflection regarding implicit bias and issues of equity and diversity. To analyze these 
findings, we turn to insights from the K-12 cultural responsive education literature (Gonzalez et 
al., 2013; Ladson-Billings, 1995; Villegas & Lucas, 2002). These frameworks promote equity- 
driven reflection that recognizes, appreciates, and attends to student differences. We argue that as 
university demographics continue to change, it becomes all the more important for scholars of 
higher education to utilize this work by incorporating it into postsecondary education research. 


Research in K-12 education has shown the importance of critical reflection for teacher 
professional development (Zeichner & Liston, 2014) and for the equitable treatment of students 
(Gonzalez et al., 2013; Ladson-Billings, 1995; Villegas & Lucas, 2002). Critical K-12 education 
frameworks, such as cultural relevant pedagogy (Ladson-Billings, 1995; Villegas & Lucas, 2002) 
and funds of knowledge (Gonzalez et al., 2013), advocate for a critical examination of personal 
assumptions, beliefs, and biases to dismantle deficit perspectives and promote more inclusive 
and culturally responsive teaching practices. We argue that these frameworks can be applied to 
our social-critical reflective practice findings to address issues of implicit bias, such as an 
“acceptable failure rate.” 


Although faculty in our sample drew on a variety of data during their reflection, the 
“reliability” of this information depended greatly on the student from which it came. Data from 
successful students was highly valued while data from unsuccessful students was discredited. 
Faculty tried to explain this distinction by discerning the students in class who were “truly 
struggling” from those who had failed because of a lack of effort, motivation, or prioritization. 
Faculty made this determination based on how seriously they perceived a student to take a course 
by using factors such as attendance and homework completion. Students who persistently failed, 
and who were not deemed “truly struggling,” were categorized as part of an “acceptable failure 
rate.” Faculty defined this “acceptable failure rate” as a certain portion of students who always 
failed because of individual shortcomings, effort, and motivation. Therefore, students who fell 
into the “acceptable failure rate” were typically those who would also be considered “unreliable” 
in terms of teaching data. 


By discrediting the experiences of certain students and framing learning as an individual 
responsibility, faculty ignored the numerous situational, contextual, and sociocultural factors 
involved in student achievement (Anyon, 1997; Bank & Banks, 2003; Gonzalez et al., 2013; 
Ladson-Billings, 1995; Villegas & Lucas, 2002). Likewise, faculty also revealed their own 
embedded assumptions of what “good” student behavior looks like without consideration of 
possible alternatives. Reliance on this stereotype presents a missed opportunity for faculty to 
engage in critical reflection to determine underlying causes and issues of student failings. Such 
critical reflection would involve a reframing of the situation that examines not only the isolated 
behavior of the student within the classroom, but the broader social and cultural worlds in which 
that student is situated. For example, instead of assuming a student has poor attendance because 
he or she does not care about the class, a more critical reflection may reveal that the student has 
poor attendance because he or she is a parent, works a full-time job, or struggles with 
transportation to school. 


17 


How Faculty Reflect on Instructional Data 


Furthermore, a more critical examination of who exactly the students are within the 
“acceptable failure rate” may reveal significant achievement gaps among student groups that 
would indicate a need for pedagogical change that responds to students’ social, cultural, and 
linguistic needs (Gonzalez et al., 2013; Ladson-Billings, 1995; Villegas & Lucas, 2002). In fact, 
Professor Sullivan was the only faculty in our sample who made the link among persistent 
educational disparities in her department, traditionally underrepresented populations at her 
institution, and outdated teaching practices. This lack of understanding indicates a need for a 
high quality data source for instructors that relates student demographic data with failure rates. 
This type of data could provide a tool to draw instructor attention to potential disparities in their 
classrooms, departments, and campuses. However, we must also be cautious and avoid using 
such tools to essentialize the experiences of underrepresented groups based on their gender, 
ethnicity, or economic background. Necessary training for instructors would be needed to 
properly read and use such data. 


Likewise, expecting faculty to have the knowledge and expertise to engage in the sort of 
critical reflection we have discussed is unreasonable, particularly if they have not been formally 
trained in teaching and learning. This also highlights the key role that institutional leaders and 
policymakers must play in training instructors to examine their implicit biases, address students’ 
individual and group needs, and recognize patterns of educational disparities. While such 
training would require deep commitment and significant resources in a time of considerable 
budget uncertainties, we argue that it is vital to push against the notion of an “acceptable failure 
rate” and address the needs of all students. Moreover, after faculty have received training on 
critical reflection, it is important for them to have the necessary space and time to collaboratively 
engage in such reflection. 


However, as structural issues like budget cuts increase, time for professional 
development to address these social issues becomes less likely. The same is true for the issues 
faculty brought up through their structural reflection. The main structural issues faculty 
discussed were oversized classes, insufficient time, and ineffective online learning systems. 
Faculty attempted to address these broader structural issues and make their workloads more 
manageable through isolated classroom workarounds. However, the coping mechanisms faculty 
developed function more like quick fixes than actual solutions. This finding indicates a 
significant need for a space for faculty to voice their concerns over such structural issues and 
explore potential solutions. Such a space would need to value faculty experience, the tacit 
knowledge they have developed as instructors, in order to be responsive to instructor and 
departmental needs. This point is particularly important as it speaks against the spread of the sort 
of top-down accountability policies that have plagued K-12 education since the early 2000s (Au, 
2009; Ravitch, 2013) to higher education. Instead we recognize faculty as adult learners who 
require additional critical reflection training, but still hold considerable knowledge as 
experienced teaching professionals. 


To conclude, our findings point to the need for a more holistic, multi-disciplinary, and 
critical understanding of DDDM and reflective practice in the field of higher education that 
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draws upon theoretical and practical insights from disciplines such as K-12 education and 
cognitive psychology. Continuing to assume that numeric data will magically transform 
educational practice is untenable given extensive evidence to the contrary (e.g., Coburn & 
Turner, 2011), and overlooking the political context of DDDM and the socio-political conditions 
of student experiences in college results in an incomplete understanding of the complex issues 
facing faculty work and student achievement in U.S. colleges and universities in the early 21“ 
century. If postsecondary institutions truly wish to grow and serve our changing population of 
learners, we must teach instructors how to educate students who may be much different from 
them and from students they previously taught. This involves teaching instructors how to 
recognize, appreciate, and utilize the cultural ways of knowing and learning students bring to the 
classroom to better facilitate learner success. Therefore, while we recognize the current 
budgetary plights and accountability pressures many universities across the country are facing, 
we Call on adult educators to advocate for their students and themselves, on state governments of 
public education institutions to show their valuing of education by fully funding and training 
postsecondary and adult instruction, and on education reformers to broaden notions of teaching 
and learning “data.” 
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